Locally Optimal and Heavy Ball GMRES Methods

The Generalized Minimal Residual method (GMRES) seeks optimal approximate solutions of linear system Ax = b from Krylov subspaces by minimizing the residual norm ‖Ax − b‖2 over all x in the subspaces. Its main cost is computing and storing basis vectors of the subspaces. For difficult systems, Krylov subspaces of very high dimensions are necessary for obtaining approximate solutions with desired accuracy. In such cases, memory requirement for using GMRES may be too demanding to be practical, and then, as an alternative, the restarted GMRES is often used instead. The latter manages to cut down memory need by simply capping the dimensions of the involved Krylov subspaces and attempts to compute approximate solutions with desired accuracy by simply repeating GMRES with the approximate solution of current GMRES cycle as the initial guess of the next GMRES cycle. The price to pay usually is slower speed of convergence. The key reason for this loss in speed is due to that in the restarted GMRES, the Krylov subspace built at each GMRES cycle is thrown away almost completely in the next cycle, except for the computed approximation solution at the cycle. How to recover the loss therefore lies in how to keep more information in Krylov subspaces built in all previous cycles and at the same time without increasing cost in computing and storing basis vectors. In this paper, we draw inspirational ideas from locally optimal conjugate gradient and multistep methods in optimization to propose two variants of the restarted GMRES. Both variants preserve the advantages of the restarted GMRES in their ability to control cost in computing and storing basis vectors, easiness to implement, and yet, as overwhelmingly numerical evidence demonstrates, are able to make up the lost information hidden in thrown-away Krylov subspaces for fast convergence.